August 15, 1990 



,'V . 



TDA Progress Report 42-102 


N9 1 - 1 1 98 0 ^ 


The Theoretical Limits of Source and Channel Coding 

S. J. Dolinar and F. Pollara 

Communications Systems Research Section 


This article presents the theoretical relationship among signal power , distortion , 
and bandwidth for several source and channel models. It is intended as a reference 
for the evaluation of the performance of specific data compression algorithms. 


I. Introduction 

The theoretical limits on the performance of source and 
channel coding are well known for several source and chan- 
nel models [1,3]. In this article these limits are calculated 
for the Gaussian channel, used as a model of the deep- 
space channel, and for some simple sources, potentially 
useful as models of planetary images. The formulas un- 
derlying these calculations are well known; the aim of this 
article is to collect and graphically display the results. The 
performance of specific data compression algorithms can 
be compared to the ultimate limits shown in these graphs. 
Similar results were presented in [2] for a binary symmetric 
source with probability of error distortion criterion. 

The results show the basic tradeoffs that must be made 
among signal power, distortion, and bandwidth, assuming 
that the system designer is free to design encoders and 
decoders but has no control over the source, the channel, 
or the user. These results also suggest that improvements 
in information transmission in future missions should be 
sought primarily through better source encoding rather 
than by pushing channel coding gain closer to its limit. 


II. The Communication System 

Consider a communication system as shown in Fig. 1. 
The source produces source symbols x with average en- 


ergy £ x at the rate of R x source symbols per second. The 
source encoder outputs bits b with average energy £b at the 
rate of Rb bits/sec. The channel encoder produces channel 
syrnbols y witli average energy £ y at the rate of R y channel 
symbols per second. The channel produces noisy outputs 
y\ the channel decoder and source decoder produce out- 
puts b and x, attempting to reproduce the values of b and 
x, respectively. 

A. Channel Models 

Channel capacity is defined as 

C(*) = max I(y\ y) (1) 

where I{y\y) is the average mutual information provided 
by the output y about the input y. The maximization is 
performed over all probability densities q(y) satisfying the 
desired cost constraint E[y‘ ] < £ y . The capacity is written 
as a function of an unspecified argument; it depends on the 
energy constraint £ y and the transition statistics p{y\y) 
that characterize the channel. 

Case Cl. For a discrete-time additive white Gaussian 
noise channel with continuous amplitude input and out- 
put, and with noise distribution jV(0,£ n )> Eq. (1) yields 
the capacity function in bits per channel symbol [3]: 

C{p) = t log 2 (l + p) 


62 



where p = £ y /£ n = 2£ y /N 0 , and N 0 /2 is the two-sided 
spectral density of the noise. 

As a comparison consider two additional channel mod- 
els obtained by constraining the input and/or the output 
of the additive white Gaussian noise channel. One chan- 
nel, Case C 3 , is obtained by restricting the channel in- 
put to binary symbols, while leaving the channel output 
unconstrained. The other case, Case C 2 , is obtained by 
constraining the channel output as well as its input to be 
binary, that is, by hard-quantizing the Gaussian channel 
output to two levels. 

Case C 2 . A hard-quantized Gaussian channel with bi- 
nary signaling and signal levels ±\f£y is a binary symmet- 
ric channel (BSC) with crossover probability e = Q(y/p ), 
where Q(x) = / x °° e“ u *f 2 du. The capacity for such a 

channel in bits per channel symbol is [ 3 ] 

c(p) = i - h[Q(Vp)] 

where 7i(e) = -e log 2 e - (1 - e) log 2 (l - c) is the binary 
entropy function. 

Case C3. For the binary input Gaussian channel, the 
capacity can be written [2] 

C(p) = 1 - £ u pog 2 (l + e- 2u )] 

where u is a random variable with distribution N(p,p ), 
and E u represents expectation over u. 

B. Source Models 

A discrete-time, continuous-amplitude stationary 
source is considered. Its rate distortion function is defined 
as 

R ( •) = min I{x\x) 

The minimization is over all conditional probability den- 
sities p(x \x) satisfying a distortion constraint E[d(. r, £)] < 
D, where d(-,*) is a distortion measure. As with channel 
capacity, i?(-) is written as a function of an unspecified 
argument; it depends on the distortion constraint D and 
on the source statistics p(x). 

Case SI. For a Gaussian memoryless source with mean 
square error (MSE) distortion constraint E[z 2 ] < £ z , 
where z = x — x, the rate distortion function can be 
expressed in bits per source symbol as [1] 


fl(tf) = ilog 2 (l/$), 0<«<1 (2) 

where 8 — £ z j£ x is the normalized MSE distortion. 

Case S 2 . As a comparison consider a memoryless binary 
symmetric source (BSS) under t lie probability of error dis- 
tortion measure, Prob(i’ ^ x) < P e . This source lias the 
rate distortion function [1] 

R(P e ) = 1 -n(Pe), 0 <P e < 1/2 

Case S3. A simple example of a source with memory is a 
stationary Gaussian source with MSE distortion constraint 
as in Case SI. In this case only a parametric form for R(8) 
is known [1]: 

< 5 ( 0 ) = — f min[0, $o(w)]</u> 

R(0) = j~ f max[0, log 2 }du 

47r V 

where 4 0 («) = £r=-oo and §5T is the nor ' 

malized autocorrelation function of the Gaussian process. 
Of particular interest is the case of a first-order Markov 
source with autocorrelation <f>(k) = £xl^ k K 0 < y < 1, 
which provides a good approximation to the autocorrela- 
tion function of actual images, as shown in the example of 
Fig. 2 . The rate distortion function for a Gauss -Markov 
source is shown in Fig. 3 for several values of y. Note 
that, at any given distortion level, the rate R varies in- 
significantly with 7 for moderate values of the correlation 
coefficient (7 < 0.5) but is reduced dramatically as 7 ap- 
proaches 1. 

III. The Source-Channel Coding Theorem 

Shannon’s channel coding theorem and source coding 
theorem can be merged into the source-channel coding 
theorem [3], which is the central result of information the- 
ory. The source-channel coding theorem answers the fol- 
lowing fundamental question: Given a source with rate 
distortion function /?(*) and a channel with capacity func- 
tion C( ), under what conditions is it possible, with suffi- 
cient coding, to achieve source distortion and channel cost 
that do not exceed the constraints used in the definitions 
of R(-) and C( ), respectively? According to the theorem, 
this is always possible if the capacity C( ) is greater than 
the rate distortion R( ) (where the two functions must be 
measured in consistent units), and never possible if the 
rate distortion is greater than the capacity. 
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In general, the source symbol rate R x need not corre- 
spond to the channel symbol rate Ry . Therefore the re- 
quirement on the information transmission system is nor- 
malized as 

C(‘)Ry > R( )Rt 

or 

C(-) > rR( ) (3) 


C(rR( )p„) > r R{ ) 

These formulas give the required relationships among r, pi, 
or p x , and the unspecified distortion variable in R( ). 

Case SI. Cl. A Gaussian channel and a Gaussian memo- 
ryless source under the MSE distortion measure gives the 
requirement 

P>6~ r - 1 (4) 


where r = R x /Ry is the number of source symbols per 
channel symbol. A large r corresponds to a small channel 
bandwidth requirement relative to the bandwidth of the 
source. To get an idea of the practical range of interest of 
r, consider the following examples: A system with an 8- 
bit/symbol source and no source compression on a channel 
encoded with code rate 1/6 yields r ~ 0.02; at another 
extreme, the same source with a source compression ratio 
of 8 : 1 and no channel coding yields r = 1; the Voyager 
concatenated system with a source compression of 2 : 1 
yields r ~ 0.11. 

This requirement can now be applied to all the combi- 
nations of source and channel models introduced in Sec- 
tion II. For all three channels considered in Section II, 
the capacity C(-) is a function of a single variable p, 
which is a signal-to-noise ratio involving the energy per 
encoded channel symbol. Usually it is more meaning- 
ful to measure the channel cost in terms of a signal-to- 
noise ratio, p b = £ b /£n = 2£b/N 0 , involving the energy 
per channel encoder input bit, or a signal-to-noise ratio, 
p x = £ x /£ n = 2£ X /N(u involving the energy per source 
symbol. The variables p, pb , and p x are related by 


Rx 


Rb 


rp x = — Px = P = Jj-Pb = r cpb = rr s p b 


Ry 


where r c = R b /R y is the rate of the channel code, and 
r s = Rb/Rx is the rate of the source code. Therefore the 
capacity constraint, Eq. (3), can be equivalently written 
in terms of p x as 


For a given source and channel the designer seeks a 
realization with small 8 and p, and large r. Of course 
these are conflicting requirements, limited by Eq. (4), and 
illustrated in Figs. 4(a), 4(b), and 4(c), where the feasi- 
ble region is above each curve with a given parameter r. 
When the source and channel have the same clock rate 
(r = 1) and p + 1 = 1/8, the source and channel are per- 
fectly matched, and ideal performance C = R is obtained 
without any need for source or channel coding, assuming 
proper scaling of the signal levels. If the channel clock rate 
is much higher than the source clock rate, the required 
£ b /No approaches the well known infinite bandwidth ex- 
pansion limit of —1.59 dB. 

Case S2.C1. For a BSS and an unconstrained Gaussian 
channel the following requirement, shown in Figs. 5(a), 
5(b), and 5(c), is obtained: 

p >2 2r[l-W(P.)J _ i 

In this case the distortion P e is extremely sensitive to small 
changes in £ b /N 0 or r. Conversely, extremely low error 
probabilities can be obtained as long as £ b /No is greater 
than the channel capacity limit for any fixed channel code 
rate, r c = r. 

Case S1.C2. A G aussian memory less source and a BSC 
gives the requirement 

2 [i“«(01 > S~ r/2 


C(rp x ) > rR ( •) 

or, in terms of p b as 

C(rr 9 pb) > rR(') 

With r s taken to equal its minimum allowable value, R(-), 
the latter requirement is 


which relates the channel cost c = Q(y/p) to th e normal- 
ized MSE distortion 6, as in Figs. 6(a), 6(b), and 6(c). 
The asymptotic behavior of these graphs for large signal- 
to-noise ratio is due to the hard quantization introduced by 
the BSC. The quantization loss can be seen by comparing 
Fig. 6(a) to Fig. 4(a), Fig. 6(b) to Fig. 4(b), or Fig. 6(c) 
to Fig. 4(c). The required £ b /N 0 at infinite bandwidth 
expansion is 0.37 dB, or 1.96 dB higher than the corre- 
sponding requirement for the unquantized channel. 
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Case S2.C2. For a BSS and BSC under the P e distortion 
measure, the requirement is 

i-n(c)>r[i-n(p e )] 

as illustrated in Figs. 7(a), 7(b), and 7(c). When r = 1 and 
e — P e } another case of perfect matching between source 
and channel is obtained, which yields ideal performance 
C — R without any need for source or channel coding. 
The loss due to the hard quantization of the channel can 
be seen by comparing Fig. 7(a) to Fig. 5(a), or Fig. 7(b) 
to Fig. 5(b), or Fig. 7(c) to Fig. 5(c). 

Case S2.C3. The cases involving the binary input Gaus- 
sian channel do not yield a simple analytical expression. 
The results are shown in Figs. 8(a), 8(b), and 8(c), for 
this channel combined with the BSS. This source-channel 
combination is the same case considered in [2], and 
Fig. 8(c) is directly comparable to Fig. 1 of [2]. (Note 
that Ef,/No in [2] refers to the source symbol signal- to- 
noise ratio denoted £ x /Nq in this article.) 


Case S3. Cl. Finally, the case of a Gauss-Markov source 
and Gaussian channel is shown in Figs. 9, 10, and 11, for 
three selected values of the rate r and several values of 
the correlation coefficient j. It is apparent that a Gauss- 
Markov source with j = 0.99, which is a good model for 
some highly correlated planetary images, ofFers large sav- 
ings in signal power due to the high redundancy of the 
source. 


IV. Conclusion 

The results presented provide a reference against which 
the performance of specific data compression algorithms 
can be measured. Choosing an appropriate model from 
Figs. 4-11, and a desired ratio r of source symbol rate 
to channel symbol rate, one can determine the minimum 
channel symbol signal- to-noise ratio £ v /Nq , bit signal-to- 
noise ratio £b/No, or source symbol signal-to-noise ratio 
£x/Nq required to produce a normalized source distortion 
that is less than /) or P e . 
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Fig. 1. A communication system model. 



ff (6) 


Fig. 2. Autocorrelation functions of a first-order Markov model 
and of an image of the moon. 


Fig. 3. Rate distortion function for a Gauss-Markov source with 
correlation 7. 
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Fig. 9. Source-channel coding limits for a Gauss-Markov source 
and a Gaussian channel (r =1): distortion versus source- 

encoded bit signal-to-noise ratio. 



Fig. 11. Source-channel coding limits for a Gauss-Markov source 
and a Gaussian channel (r =1/16): distortion versus source- 
encoded bit signal-to-noise ratio. 
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Fig. 10. Source-channel coding limits for a Gauss-Markov source 
and a Gaussian channel (r = 1/4): distortion versus source- 
encoded bit signal-to-noise ratio. 






